Color image processing apparatus

ABSTRACT

A color image processing apparatus reproduces color data of a subject at higher accuracy by estimating the spectral reflectance of the subject from the color chip basis function vn(λ) which can expand the spectral reflectances of all color chips of the reference chart and the expansion coefficients B of the color chips. The color image processing apparatus uses an image processing device to which a subject shooting signal Q shot by the color image input device and a reference chart shooting signal G taken from a plurality of color chips each having a spectral reflectance, is known.

This is a division of application Ser. No. 10/395,350 filed Mar. 24,2003, now U.S. Pat. No. 7,010,162, which is a division of applicationSer. No. 09/152,974 filed on Sep. 14, 1998, now U.S. Pat. No. 6,549,653issued on Apr. 15, 2003.

BACKGROUND OF THE INVENTION

The present invention relates to a color image processing apparatuswhich estimates the spectral reflectance of an object to be shot, from acolor image signal.

As the color imaging apparatus becomes more popular, it becomes moreimportant to reproduce the original color of an object exactly using animage display apparatus such as a CRT monitor or a printing device suchas a printer, from a color image signal acquired by a color image signalinput device (to be referred to simply as an “input device” hereinafter)such as a color scanner or a digital camera.

Further, as networks connecting a plurality of devices via wiring arewidely used, the accurate transmission of the color data of a colorimage signal and reproduction of color are becoming more importanttechnical factors for establishing systems for remote medical check-upsand examinations in the field of telemedicine or electronic shopping,which are expected to significantly expand from now.

In order to establish a system capable of reproducing an accurate color,it is firstly required to estimate the color data of an objectaccurately from a color image signal obtained from an input device.

As a method of dealing with color quantitatively, the values of XYZcolor system which is represented by the following equation (1) usingfunctions of wavelength λ, x(λ), y(λ) and z(λ), corresponding to thespectral sensitivities of human vision, which is called as colormatching function as defined by Commission International de I'Eclairage(CIE), and the spectrum of an object w(λ), or a uniform color spacebased on the XYZ color system are widely used.

$\begin{matrix}{{X = {K{\int_{\lambda = 380}^{780}{{x(\lambda)}{w(\lambda)}{\mathbb{d}\lambda}}}}}{Y = {K\ {\int_{\lambda = 380}^{780}{{x(\lambda)}{w(\lambda)}{\mathbb{d}\lambda}}}}}Z = {K{\int_{\lambda = 380}^{780}{{z(\lambda)}{w(\lambda)}{\mathbb{d}\lambda}}}}} & (1)\end{matrix}$

In the above equation, K is a constant. In order to obtain color data ofan object accurately from a color image signal, it is essential toaccurately estimate XYZ values of an object under illumination ofobservation-site from the color image signal. In the case where theillumination used for acquiring the color image signal and theillumination of observation-site are equal to each other, the problem isto estimate the XYZ values of the object under the illumination used foracquiring the color image signal.

In this case in order to accurately obtain the XYZ values of anarbitrary object under the illumination used for obtaining a color imagesignal, it is required that the spectral sensitivities of the inputdevice and the color matching functions should have a linear conversionrelationship.

It is well known that the color image signal of an arbitrary object canbe accurately converted to XYZ values by linear conversion only if theabove condition called a Luther condition is satisfied.

The conversion relationship can be obtained from the relationshipbetween the spectral sensitivities of the input device and the colormatching functions; however it can be indirectly obtained from therelation between color image signals of three or more objects havingindependent XYZ values, and measured XYZ values.

Usually, it is very difficult to accurately measure the spectralsensitivities of the input device, and therefore the conversionrelationship from color image signals to XYZ values is usually obtainedby the latter method.

In the case where the spectral sensitivities of the input device do notsatisfy the Luther condition in a strict sense, accurate XYZ valuescannot be obtained from a color image signal for an arbitrary object.However, with regard to the specific object, the conversion relationshipcan be obtained by a way of least square method from the relationshipbetween the color image signals and XYZ values for a number of colors ofobjects.

The format of the color chart used as the object for obtaining colordata from a color image signal acquired by the input device, isstandardized by ISO IT8.7, and provided by several film makers.

In order to obtain the color data of an object under illumination ofobservation-site different from the illumination of shooting-site usedfor acquiring a color image signal with respect to an arbitrary object,the product of the spectral sensitivities of the input device and theshooting-site illumination spectrum, and the product of the colormatching functions and the observation-site illumination spectrum musthave the relationship of linear conversions.

Such a condition depends upon the illumination spectrum, and thereforeit is not practical in general. Consequently, in order to estimate thecolor data of an object under illumination different from that ofshooting-site, it is necessary to obtain the spectral reflectance of theobject.

For an accurate estimation of the spectral reflectance of an object, themethod of obtaining the spectral reflectance of an object by estimatingthe spectrum from a lot of multi-channel images acquired by the spectralsensitivities of narrow bands, and dividing the estimated spectrum byillumination spectrum can be proposed as disclosed in, for example, Jpn.Pat. Appln. KOKAI Publication No. 9-172649.

Further, as disclosed in the Journal of Imaging Science and TechnologyVol. 40, No. 5, Sep./Oct. 1996, p422–p430, in the case where objects tobe shot are limited to particular subjects such as human skin and thespectral reflectance is represented by a linear combination of a smallnumber of basis functions, it becomes possible to estimate the spectralreflectance from color image signals the band number of which is equalto or more than the number of basis functions representing the spectralreflectance of the object.

In these methods, the data of the spectrum of the illumination used toacquire the color image signal is necessary in addition to the spectralsensitivities of the input device.

However, it is difficult for the operator to accurately measure thespectral sensitivities of the input device. In consideration of adifference between individual input devices, change along with time orthe like, the accuracy of the data provided by the maker of the deviceis not always sufficient.

Further, in order to obtain the data of the spectrum of the illuminationfor the shooting environment, a measurement device such as aspectrophotometer is required when shooting, and therefore these methodcannot be easily applied to a system which employs a conventional colorimage input device.

BRIEF SUMMARY OF THE INVENTION

The object of the present invention is to provide a color imageprocessing device capable of estimating the spectral reflectance of anobject to be shot from the color image signal of the object acquired bya color image signal input device, even if the spectral sensitivities ofthe color image signal input device, or the spectrum of the illuminationused when the color image signal is input are unknown.

According to the present invention, there is provided a color imageprocessing device including image processing means for subjecting animage signal input of an object to be shot, to a predetermined imageprocessing, so as to be able to reproduce color data of the object,wherein the image processing means carries out the process ofreproducing the color data of the object, by estimating the spectralreflectance of the object using the color image signal of a referenceobject the spectral reflectance of which is known, the color imagesignal of an object acquired by the same illumination condition as thatfor acquiring the color image signal of the reference object, and thespectral reflectance data of the reference object.

Additional objects and advantages of the invention will be set forth inthe description which follows, and in part will be obvious from thedescription, or may be learned by practice of the invention. The objectsand advantages of the invention may be realized and obtained by means ofthe instrumentalities and combinations particularly pointed outhereinafter.

BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWING

The accompanying drawings, which are incorporated in and constitute apart of the specification, illustrate presently preferred embodiments ofthe invention, and together with the general description given above andthe detailed description of the preferred embodiments give below, serveto explain the principles of the invention.

FIG. 1 is a diagram showing a structure of a color image processingdevice according to the first embodiment of the present invention;

FIG. 2 is a diagram showing a detailed structure of an image processingdevice in the color image processing device according to the firstembodiment;

FIG. 3 is a diagram showing a structure of an object expansioncoefficient calculating unit shown in FIG. 2;

FIG. 4 is a diagram showing a structure of a spectral reflectancecalculating unit shown in FIG. 2;

FIG. 5A is a diagram showing a structure of the color image input deviceshown in FIG. 1;

FIG. 5B is a diagram showing a structure of a filter turret to whichinterference filters are mounted and which are built in the color imageinput device;

FIG. 6 is a diagram showing a structure of an RGB value calculating unitshown in FIG. 2;

FIG. 7 is a diagram showing an example of a reference chart used in thepresent invention;

FIG. 8 is a diagram showing a structure of a color image processingdevice according to the second embodiment of the present invention;

FIG. 9 is a diagram showing a detailed structure of a image processingdevice in the color image processing device according to the secondembodiment;

FIG. 10 is a diagram showing a structure of an object expansioncoefficient calculating unit shown in FIG. 9;

FIG. 11 is a diagram showing a structure of a spectral reflectancecalculating unit shown in FIG. 9;

FIG. 12 is a diagram showing a structure of a color image processingdevice according to the third embodiment of the present invention;

FIG. 13 is a diagram showing a detailed structure of an image processingdevice in the color image processing device according to the thirdembodiment;

FIG. 14 is a diagram showing a structure of an object expansioncoefficient calculating unit shown in FIG. 13;

FIG. 15 is a diagram showing a structure of a color image processingdevice according to the fourth embodiment of the present invention;

FIG. 16 is a diagram showing a structure of a spectral reflectanceevaluating unit shown in FIG. 15;

FIG. 17 is a diagram showing a structure of a color image processingdevice according to the fifth embodiment of the present invention;

FIG. 18 is a diagram showing a concept of a format of shot image datastored on a memory card; and

FIG. 19 is a diagram illustrating a color conversion process in thefifth embodiment.

DETAILED DESCRIPTION OF THE INVENTION

Embodiments of the present invention will now be described withreference to drawings.

First, the concept of a color image processing device according to thepresent invention will be provided.

According to the present invention, the spectral reflectance of anobject is estimated from the spectral reflectance of reference objectswhose spectral reflectances are known a priori and color image signalvalues of the reference objects and color image signal values of theobject whose spectral reflectance is not known acquired from the colorimage signal input device (to be called simply “input device”hereinafter).

A method of estimating the spectral reflectance of an object from acolor image, using such an input device will now be described inconnection with an example case of a camera having the L-number ofindependent spectral sensitivities (bands).

Let us suppose here that the spectral sensitivity of the k-th band ofthe camera is represented by h^(k)(λ) (k=1 to L), the spectrum of theillumination for shooting is represented by S(λ), and the spectralreflectance of the object is represented by p(λ), and p(λ) can beexpanded by the L-number of basis functions el(λ) (l=1 to L).

A signal value q^(k) obtained by shooting an object under theshooting-site illumination at the k-th camera sensitivity, is given bythe following formula supposing that the sensitivity of the camera makesa linear response to the intensity of incident light:q ^(k) =∫p(λ)s(λ)h ^(k)(λ)dλ  (2)

Here, since the spectral reflectance p(λ) of an object can be expandedinto the L-number of basis functions el(λ) (l=1 to L), p(λ) is given byformula (3) using expansion coefficients cl (l=1 to L).

$\begin{matrix}{{p(\lambda)} = {\sum\limits_{l = 1}^{L}\;{c_{l}{e_{l}(\lambda)}}}} & (3)\end{matrix}$Therefore, formula (2) can be rewritten as:

$\begin{matrix}{q^{k} = {\sum\limits_{l = 1}^{L}\;{c_{l}a_{l}^{k}}}} & (4)\end{matrix}$expect that:a _(l) ^(k) =∫e _(l)(λ)s(λ)h ^(k)(λ)dλ  (5)

The signal values represented by equation (4) are obtained for theL-number of the sensitivities of the camera, the following matrixequation (6) is given:

$\begin{matrix}{\begin{pmatrix}q^{1} \\q^{2} \\\ldots \\q^{L}\end{pmatrix} = {\begin{pmatrix}a_{1}^{1} & a_{2}^{1} & \ldots & a_{L}^{1} \\a_{1}^{2} & a_{2}^{2} & \ldots & a_{L}^{2} \\\ldots & \; & \; & \; \\a_{1}^{L} & a_{2}^{L} & \ldots & a_{L}^{L}\end{pmatrix}\begin{pmatrix}c_{1} \\c_{2} \\\ldots \\c_{L}\end{pmatrix}}} & (6)\end{matrix}$

The matrices of the equation (6) are represented by Q, A and C,respectively to give:Q=AC  (7)

Then, the estimated vector C of the expansion coefficient cl (l=1 to L)of each basis function of spectral reflectance of the object is given bythe following formula.C=A ⁻¹ Q  (8)

In the formula (8), Q is a matrix known by measurement, and thereforewhen A is available, the value for C can be obtained. When C isobtained, it becomes possible to obtain the spectral reflectance p(λ) ofthe object from the formula (3).

In order to obtain the value for A, the M-number of color chips whosespectral reflectances are known are shot. Suppose that the spectralreflectance of the m-th color chip is represented by fm(λ) (m=1 to M).If the spectral reflectance fm(λ) of the color chips can be expanded bythe M-number of basis functions vn(λ) (n=1 to M), fm(λ) can be givenwith use of expansion coefficient bmn (n=1 to M), by:

$\begin{matrix}{{f_{m}(\lambda)} = {\sum\limits_{n = 1}^{M}\;{b_{mn}{v_{n}(\lambda)}}}} & (9)\end{matrix}$

The shooting signal value g^(k)m of the m-th color chip by the k-thcamera sensitivity, is given by:g _(m) ^(k) =∫f _(m)(λ)s(λ)h ^(k)(λ)dλ  (10)

From the formulas (9) and (10), the formula (11) is given by:

$\begin{matrix}{g_{m}^{k} = {\sum\limits_{n = 1}^{M}\;{b_{mn}d_{n}^{k}}}} & (11)\end{matrix}$expect that:d _(n) ^(k) =∫v _(n)(λ)s(λ)h ^(k)(λ)dλ  (12)

The signal value represented by equation (11) is obtained for theL-number of the sensitivities of the camera, the following matrixequation (13) is given:

$\begin{matrix}{\begin{pmatrix}g_{1}^{1} & g_{2}^{1} & \ldots & g_{M}^{1} \\g_{1}^{2} & g_{2}^{2} & \ldots & g_{M}^{2} \\\ldots & \; & \; & \; \\g_{1}^{L} & g_{2}^{L} & \ldots & g_{M}^{L}\end{pmatrix} = {\begin{pmatrix}d_{1}^{1} & d_{2}^{1} & \ldots & d_{M}^{1} \\d_{1}^{2} & d_{2}^{2} & \ldots & d_{M}^{2} \\\ldots & \; & \; & \; \\d_{1}^{L} & d_{2}^{L} & \ldots & d_{M}^{L}\end{pmatrix}\begin{pmatrix}b_{11} & b_{21} & \ldots & b_{M1} \\b_{12} & b_{22} & \ldots & b_{M2} \\\ldots & \; & \; & \; \\b_{1M} & b_{2M} & \ldots & b_{MM}\end{pmatrix}}} & (13)\end{matrix}$

The matrices of the equation (13) are represented by G, D and B,respectively to give:G=DB  (14)

From the equation (14), D can be given by:D=GB ⁻¹  (15)

In the formula (15), since G is a measurement data and B is an expansioncoefficient data corresponding to the basis functions of thepre-obtained spectral reflectances of the color chips, D can beobtained.

Next, when the basis function el(λ) (l=1 to L) of the spectralreflectance of the object is expanded by the basis functions vn(λ) (n=1to M) of the spectral reflectances of the color chips, the basisfunction el(λ) can be represented by the following formula using theexpansion coefficient Oln.

$\begin{matrix}{{{e_{l}(\lambda)}{\sum\limits_{n = 1}^{M}\;{o_{\ln}{v_{n}(\lambda)}}}} + {\delta_{l}(\lambda)}} & (16)\end{matrix}$

In the above equation, δl(λ) is an expansion error. When the formula(16) is substituted with the formula (5), the following formula isgiven:

$\begin{matrix}\begin{matrix}{a_{l}^{k} = {\int{\left\{ {{\sum\limits_{n = 1}^{M}\;{o_{\ln}{v_{n}(\lambda)}}} + {\delta_{l}(\lambda)}} \right\}{s(\lambda)}{h^{k}(\lambda)}{\mathbb{d}\lambda}}}} \\{= {{\sum\limits_{n = 1}^{M}\;{o_{\ln}{\int{{v_{n}(\lambda)}{s(\lambda)}{h^{k}(\lambda)}{\mathbb{d}\lambda}}}}} + {\int{{\delta_{l}(\lambda)}{s(\lambda)}{h^{k}(\lambda)}{\mathbb{d}\lambda}}}}} \\{= {{\sum\limits_{n = 1}^{M}\;{o_{\ln}d_{n}^{k}}} + \Delta_{l}^{k}}}\end{matrix} & (17)\end{matrix}$

If the formula (17) is represented as:A=DO+Δ  (18)

In this equation, if term Δ created by expansion error δl(λ) can beneglected, the estimated vector A′ of A is given by:A′=DO  (19)

The estimated vector C′ of the expansion coefficient vector C of eachbasis function of the spectral reflectance of the object is given by:

$\begin{matrix}\begin{matrix}{C^{\prime} = {{A^{\prime}}^{- l}Q}} \\{= {({DO})^{- 1}Q}} \\{= {\left( {{GB}^{- 1}O} \right)^{- 1}Q}}\end{matrix} & (20)\end{matrix}$

In this formula (20), Q represents a shooting signal of an object, Orepresents an expansion coefficient obtained by expanding the basisfunctions of the spectral reflectance of the object by the basisfunctions of the spectral reflectances of the color chips, B representsthe expansion coefficient of the basis function of each spectralreflectance of the color chips, and G represents a shooting signal ofthe color chips.

By shooting a color chips whose spectral reflectances are known, G isobtained. Thus, from the formula (20), with values of an expansioncoefficient B of the basis functions of each spectral reflectance of thecolor chips which are obtained in advance, an expansion coefficient O ofthe basis functions of the spectral reflectances of an object which iscalculated in advance and a shooting signal Q of an object, theexpansion coefficient of the basis functions for the spectralreflectance of an object can be obtained. Then, from the basis functionsfor the spectral reflectance of the object and the expansion coefficientC, the spectral reflectance of the object is calculated.

In particular, when the basis functions el(λ) of the spectralreflectance of an object and the basis functions vn(λ) of color chipscoincide with each other, O becomes a unit matrix, and the estimatedvector C′ of C is given by:C′=(GB ⁻¹)⁻¹ Q  (21)

As represented by the formula (21), when the basis functions for thespectral reflectances of the color chips and the basis function of thespectral reflectance of the object coincide with each other, thespectral reflectance of the object can be obtained from the shootingsignal Q of the signal using a color chip shooting image signal G shotby a camera having the same number of bands as the number of the basisfunctions.

When the spectral reflectance of an object can be expanded by the basisfunctions of the color chips, the spectral reflectance of the object canbe obtained as a linear combination of the basis functions of thespectral reflectances of the color chips, by rendering C′ of the formula(21) the expansion coefficient of the basis functions of the spectralreflectance of the color chips.

In this case, even if the basis functions of the spectral reflectance ofan object is not known, the spectral reflectance of an object can beobtained by a camera having bands whose number is equal to or more thanthat of the basis functions of the spectral reflectances of the colorchips. Further, if the basis functions of the spectral reflectances ofthe object are known, and it can be expanded by the basis functions ofthe spectral reflectances of the color chips, the spectral reflectanceof the object can be obtained by the formula (20) from the shootingsignal Q of the object, using the color chip shooting image signal Gtaken by a camera having bands whose number is equal to or more thanthat of the basis functions of the object.

Next, it is the case where different illumination conditions are usedfor the shooting of the color chips and the object.

When the illumination spectrum at the shooting of an object is So(λ),the formula (2) can be rewritten as:q ^(k) =∫p(λ)s _(o)(λ)h ^(k)(λ)dλ  (22)

Further, the formula (5) can be expressed as:a _(l) ^(k) =∫e _(l)(λ)s _(o)(λ)h ^(k)(λ)dλ  (23)

When the illumination spectrum at the shooting of color chips is Sc(λ),the formula (10) can be rewritten as:g _(m) ^(k) =∫f _(m)(λ)s _(c)(λ)h _(k)(λ)dλ  (24)

Further, the formula (12) can be expressed as:d _(n) ^(k) =∫v _(n)(λ)s _(c)(λ)h ^(k)(λ)dλ  (25)

Whene′ _(l)(λ)=e _(l)(λ)s _(o)(λ)  (26)v′ _(n)(λ)=v _(n)(λ)s _(c)(λ)  (27)and el′(λ) is expanded by vn′(λ), el′(λ) can be represented, with use ofexpansion coefficient O′ln, as:

$\begin{matrix}{{e_{l}^{\prime}(\lambda)} = {{\sum\limits_{n = 1}^{M}{o_{l\; n}^{\prime}{v_{n}^{\prime}(\lambda)}}} + {\delta_{l}^{\prime}(\lambda)}}} & (28)\end{matrix}$

From the formulas (28) and (23),

$\begin{matrix}\begin{matrix}{a_{l}^{k} = {\int{{e_{l}(\lambda)}{s_{o}(\lambda)}{h^{k}(\lambda)}{\mathbb{d}\lambda}}}} \\{= {\int{{e_{l}^{\prime}(\lambda)}{h^{k}(\lambda)}{\mathbb{d}\lambda}}}} \\{= {\int{\left\{ {{\sum\limits_{n = 1}^{M}{o_{l\; n}^{\prime}{v_{n}^{\prime}(\lambda)}}} + {\delta_{l}^{\prime}(\lambda)}} \right\}{h^{k}(\lambda)}{\mathbb{d}\lambda}}}} \\{= {{\sum\limits_{n = 1}^{M}{o_{l\; n}^{\prime}{\int{{v_{n}^{\prime}(\lambda)}{h^{k}(\lambda)}{\mathbb{d}\lambda}}}}} + {\int{{\delta_{l}^{\prime}(\lambda)}{h^{k}(\lambda)}{\mathbb{d}\lambda}}}}} \\{= {{\sum\limits_{n = 1}^{M}{o_{l\; n}^{\prime}d_{n}^{k}}} + \Delta_{l}^{\prime}}}\end{matrix} & (29)\end{matrix}$is given. The formula (29) can be rewritten as:A=DO′+Δ  (30)

In this equation, if term Δ′ created by expansion error δ′l(λ) can beneglected, the estimated vector A′ of A is given by:A′=DO′  (31)

The estimated vector C′ of the expansion coefficient vector C of eachbasis function of the spectral reflectance of the object is given by:

$\begin{matrix}\begin{matrix}{C^{\prime} = {A^{\prime - 1}Q}} \\{= {\left( {D\; O^{\prime}} \right)^{- 1}Q}} \\{= {\left( {G\; B^{- 1}O^{\prime}} \right)^{- 1}Q}}\end{matrix} & (32)\end{matrix}$

In this formula (32), Q represents a shooting signal of an object, O′represents an expansion coefficient obtained by expanding the product ofthe basis functions of the spectral reflectance of an object and theobject shooting-site illumination spectrum, of the formula (28) by theproducts of the basis functions of the color chips and the color chipshooting-site illumination spectrum, B represents the expansioncoefficient of the basis function of each spectral reflectance of thecolor chips, and G represents a shooting signal of the color chips.

By shooting color chips whose spectral reflectances are known, G isobtained. Thus, from the formula (32), with values of an expansioncoefficient B of the basis functions of each spectral reflectance of thecolor chips which are obtained in advance, a pre-calculated O′ and ashooting signal Q of an object, the expansion coefficients of the basisfunctions for the spectral reflectance of an object can be obtained.

Then, the spectral reflectance of the object can be calculated from itsbasis functions and expansion coefficients.

With the present invention, even in the case where the spectralsensitivities of the camera and the spectrum of the shooting-siteillumination are unknown, the spectral reflectance of an object isestimated from the shooting signal of the object, having the basisfunctions of the spectral reflectance, by which the basis function ofthe spectral reflectance of the object can be expanded.

Further, even in the case where the illumination condition for shootingthe color chips and the illumination condition for shooting the objectare different from each other, the spectral reflectance of the objectcan be estimated from the shooting signal of the object if the colorchip shooting-site illumination spectrum and object shooting-siteillumination spectrum are known.

In the estimation of the spectral reflectance of an object describedabove, there are conditions which the spectral reflectances of colorchips in reference object whose spectral reflectance are known mustsatisfy in order to accurately estimate the spectral reflectance of theobject. By forming in advance an object which satisfies theseconditions, more accurate estimation of the spectral reflectance can beperformed than in the case where a conventional reference object isused.

In connection with the above-described spectral reflectance estimatingmethod, it is explained that the number of basis functions of thespectral reflectances of the color chips whose spectral reflectance areknown, coincides with that of color chips; however a method similar tothis can be applied to cases where they are different.

Further, it is supposed in the above description that the number ofbasis functions of spectral reflectances of an object and the number ofbands of the input device coincide with each other; however this methodcan be expandable for cases where these numbers are different from eachother. For example, as in the case of an ordinary digital camera, themethod can be applied to a 3-band input device.

As a reference object whose spectral reflectance is known, the colorchip has been discussed as an example; however arbitrary objects can beapplied to this method as long as they have spectral reflectancessimilar to those of the color chips.

FIG. 1 is a diagram showing an example of the structure of a color imageprocessing apparatus according to the first embodiment of the presentinvention, which will now be described.

The apparatus of this embodiment, includes a color image input device(input device) 1, a reference chart 2 and an object 3, arranged underthe same illumination A, an observation-site illumination spectrum datastorage unit 4, an image processing device 5 and a CRT monitor 6.

The image processing device 5 shoots the reference chart 2 and theobject 3 arranged under the same illumination A in order or at the sametime by the input device 1. Further, the spectral reflectance and XYZvalues of the object 3 are estimated from the image signals G and Qoutput, and the basis functions and expansion coefficients data of thereflectances of color chips, and color matching functions data, whichare present in advance within the image processing device, and thus RGBsignals are output. The RGB signals are displayed on the CRT monitor 6as a color image of the object.

In this embodiment, the XYZ values under the observation-siteillumination are obtained from the spectral reflectance of the object,which is estimated by the image processing device 5, and further thedata is displayed on the CRT monitor as a color image. However, thepresent invention is not limited to this embodiment, but is applicableto the system including display devices such as a liquid crystal displaydevice and a plasma display, and a printing device such as a colorprinter. Further, it is possible to store the spectral reflectance andXYZ values as data.

With the color image processing apparatus having the above-describedstructure, if the reference chart 2 and the object 3 are shot at thesame time by the input device 1, the image signal of the reference chart2 is extracted from the object shooting signal by a reference chartextracting device (not shown) in the image processing device 5.

The input device 1 forms an image on a semiconductor imaging device suchas a CCD 104, from the transmitted light through a camera lens 1 and oneof a plurality of interference filters 103 mounted on the filter turret102.

These interference filters 103 are mounted on the filter turret 102having a circular shape, as schematically shown in FIG. 5B. As thefilter turret 102 is rotated by the motor 105, each of the interferencefilters 103 is mounted on the rear surface of the camera lens 101 inorder, and an image of the corresponding band is shot by the CCD 104.The CCD 104 has the pixel number of, for example, 640 pixels×480 pixels.Naturally, the present invention is not limited to this pixel number.

The output signal of an image shot by the CCD 104 is sent to the imageprocessing device 5.

FIG. 7 shows an example of the reference chart 2. The reference chart 2consists of 10 color chips, and the spectral reflectance of each colorchip can be expanded by 10 basis functions which are independent fromeach other. The image signals G and Q acquired by the input device 1 areinput to the image processing device 5 as 10 channels image data havinga pixel size of 640 pixels×480 pixels of 10-bits per pixel.

TO the image processing device 5 to which the image signals G and Q havebeen input, observation-site illumination spectrum data is input fromthe observation-site illumination spectrum data storage unit 4. Theobservation-site illumination spectrum data consists of light intensityvalues of observation-site illumination taken at an interval of 1 nm, ina wavelength range from 380 nm to 780 nm.

In the image processing device 5, the spectral reflectance and the XYZvalues of the object are estimated from these input data, and the colorchip basis function data, color chip expansion coefficient data andcolor matching functions data, which are pre-stored in the imageprocessing device. Further, the image processing device 5 converts, onthe basis of the characteristics data of the CRT monitor 6, the XYZvalues of the object 3 taken under the observation-site illuminationinto RGB signals to be displayed on the monitor screen, and outputs thesignals to the CRT monitor 6.

As the RGB signals output from the image processing device are input tothe CRT monitor 6, the CRT monitor 6 displays the color image of theobject.

Next, the structure of the image processing device 5 of this embodimentwill now be described with reference to FIG. 2.

The image processing device 5 consists mainly of a spectral reflectancecalculating unit 7, a tristimulus value calculating unit 8 and an RGBvalue calculating unit 9.

The spectral reflectance calculating unit 7 comprises a color chipshooting signal storage unit 11, a color chip expansion coefficientstorage unit 12, an object expansion coefficient calculating unit 10, acolor chip basis function storage unit 14 and a spectral reflectancecalculating unit 13 for calculating a spectral reflectance p(λ).

The color chip shooting signal storage unit 11 calculates, from theshooting image of the reference chart containing a plurality of colorchips, the average values of shooting signal values corresponding to thecolor chips, for an appropriate region, and stores them as color chipshooting signals. The color chip expansion coefficient storage unit 12stores expansion coefficients corresponding to basis functions of thespectral reflectances of the color chips in the reference chart, whichare obtained in advance.

The object expansion coefficient calculating unit 10 calculates anexpansion coefficient C of the spectral reflectance of an object fromthe object shooting signal input from the color image input device 1,the color chip shooting signal G input from the color chip shootingsignal storage unit 11 and the expansion coefficient B input from thecolor chip expansion coefficient storage unit 12.

Further, the color chip basis function storage unit 13 stores color chipbasis functions vn(λ) (n=1 to 10), and the spectral reflectancecalculating unit 13 calculates the spectral reflectance p(λ) of anobject from the expansion coefficient C calculated by the objectexpansion coefficient calculating unit 10 and the basis functions vn(λ)(n=1 to 10) of the object output from the color chip basis functionstorage unit 14.

FIG. 3 shows the structure of the object expansion coefficientcalculating unit 10, which will now be described.

An inverse matrix B⁻¹ of the expansion coefficient B calculated by aninverse matrix operating unit A19, from the expansion coefficient Binput from the color chip expansion coefficient storage unit 12, and acolor chip shooting signal G input from the color chip shooting signalstorage unit 11 are input to a matrix multiplication unit A20, and thusa matrix (GB⁻¹) is calculated. The calculated GB⁻¹ is input to aninverse matrix operating unit B21 to obtain an inverse matrix thereof(GB⁻¹)⁻¹, which is input to a matrix multiplication unit B22. To thematrix multiplication unit B22, the object shooting signal Q is furtherinput, and the expansion coefficient C=(GB⁻¹)⁻¹Q is calculated from(GB⁻¹)⁻¹ output from the inverse matrix operating unit B21.

FIG. 4 shows the structure of the spectral reflectance calculating unit13, which will now be described.

The spectral reflectance calculating unit 13 consists of a basiscoefficient multiplication unit 23 for multiplying an object expansioncoefficient C corresponding to a color chip basis function vn(λ), and abasis function integrating unit 24 for calculating the spectralreflectance.

The basis coefficient multiplication unit 23 obtains, by multiplication,the product of object expansion coefficients C=Cn (n=1 to 10)corresponding to the color chip basis functions vn(λ), from the objectexpansion coefficient C input from the object expansion coefficientcalculating unit 10 and the color chip basis function vn(λ) input fromthe color chip basis function storage unit 14, and thus obtained resultis output to the basis function integrating unit 24.

The basis function integrating unit 24 adds up the data input from thebasis coefficient multiplication unit 23 at all the wavelengths in termsof all the basis functions cnvn(λ) (n=1 to 10), and thus the spectralreflectance is calculated.

In the tristimulus value calculating unit 8 shown in FIG. 2, thetristimulus value operating unit 16 calculates the tristimulus valuesXYZ under the illumination condition for observing an object, from thespectral reflectance of the object, calculated from the spectralreflectance calculating unit 7, the observation-site illumination inputfrom the observation-site illumination spectrum data storage unit 4, andthe color matching function data pre-stored in the color matchingfunction data storage unit 15, and the values are output to the RGBvalue calculating unit 9.

The RGB value calculating unit 9, as shown in FIG. 6, comprises a CRTmonitor profile storage unit 17 for storing the characteristics data ofthe display device and an output signal calculating unit 18 forconverting the XYZ values to the monitor input signal RGB values usingthe characteristics data of the CRT monitor.

The characteristics data of the CRT monitor contains a matrix M storedin the phosphor tristimulus value data storage unit 25 and a gamma curveγ^(R)γ^(G)γ^(B) stored in the RGB gradation characteristics data storageunit 26. The matrix M is a matrix made of elements of 3×3, and the gammacurves γ^(R)γ^(G)γ^(B) is the output luminance value to each of the RGBinput values.

In the RGB value calculating unit 9, the XYZ values calculated by thetristimulus value calculating unit 8 and the matrix M stored in thephosphor tristimulus value data storage unit 25 are multiplied by thematrix multiplication unit 27. Then, the gradation correcting unit 28performs the correction of the gradation on the basis of the result ofthe multiplication and the gamma curves stored in the RGB gradationcharacteristics data storage unit 26, and thus the RGB signals areobtained by conversion.

Thus, the calculated RGB signals are input to the CRT monitor, and acolor image of the object is displayed on the CRT monitor.

According to the first embodiment of the present invention, the spectralreflectance of an object is estimated from the object shooting signal Qand reference chart shooting signal G shot under the same illuminationcondition by the color image input device, and the color chip basisfunctions vn(λ), which can expand the spectral reflectance of thespectral reflectance of all the color chips of the reference chartpre-stored, (that is, the spectral reflectance data of the object) andthe expansion coefficients B of the color chips. Here, the conditionsfor being able to estimate the spectral reflectance of an object exactlyare that the spectral reflectance of the object can be expanded by thecolor chip basis functions vn(λ) of the reference chart and the colorimage input device has the number of bands which is equal to or morethan that of color chip basis functions. With these conditions, thespectral reflectance of an object can be estimated even if the spectralsensitivities of the color image input device and the illuminationspectrum used for shooting are unknown.

FIG. 8 shows a color image processing apparatus according to the secondembodiment of the present invention.

The apparatus of this embodiment consists of an input device 1 to whicha color image is input by shooting, a reference chart 2 and an object 3,which are arranged under the same illumination A, an observation-siteillumination spectrum data storage unit 4 for storing observation-siteillumination spectrum data, an object basis function storage unit 29 forstoring object basis function el(λ), an image processing device 30 foroutputting RGB signals and a CRT monitor 6 for displaying a color image.

The image processing device 30 shoots the reference chart 2 and object3, which are exposed by the same illumination A sequentially or at thesame time, by the input device 1, and estimates the spectral reflectanceand XYZ values of the object 3 from the shooting signals G and Q ofthose shot respectively, and the color chip basis functions data, colorchip expansion coefficients data and color matching functions data,pre-stored in the image processing device. Thus, the RGB signals areoutput. The RGB signals are displayed on the CRT monitor 6 as a color(RGB) image of the object.

The input device 1 is equivalent to the input device of the firstembodiment described above, and a filter turret on which interferencefilters are arranged, as shown in FIGS. 5 and 5B, is mounted. Further,the reference chart of this embodiment has the same structure as shownin FIG. 7. Therefore, the explanations therefor will not be repeated.

To the image processing device 30, the image signals G and Q shot by theinput device 1 are input, the observation-site illumination spectrumdata is input from the observation-site illumination spectrum datastorage unit 4, and the basis function data of the object is input fromthe object basis function storage unit 29. The observation-siteillumination spectrum data consists of intensity values ofobservation-site illumination taken at an interval of 1 nm, in awavelength range from 380 nm to 780 nm.

In the image processing device 30, the spectral reflectance and XYZvalues of the object are estimated from these input data, the color chipexpansion coefficient B pre-stored in the image processing device, theexpansion coefficient O obtained by expanding the basis function of anobject by the basis functions of the color chips, the color chip basisfunctions vn(λ), and the color matching functions data. Further, RGBsignals to be input to the CRT monitor, are output; the RGB signals areused for displaying the XYZ values of the object taken under theobservation-site illumination, on the CRT monitor, using thecharacteristics data of the CRT monitor. The CRT monitor displays acolor image of the object, as the RGB signal output from the imageprocessing device is input thereto.

The structure of the image processing device will now be described indetail.

As shown in FIG. 9, the image processing device 30 consists mainly of aspectral reflectance calculating unit 31, a tristimulus valuecalculating unit 8 and an RGB value calculating unit 9. Here, thestructural elements of this embodiment, which are similar to those shownin FIG. 2, will be designated by the same reference numerals, and thedetailed descriptions therefor will not be repeated.

The spectral reflectance calculating unit 31 includes a color chipshooting signal storage unit 11 for storing a color chip shootingsignal, a color chip expansion coefficient storage unit 12 for storingan expansion coefficient B, a color chip basis function storage unit 14for storing color chip basis functions vn(λ) (n=1 to 10), an objectbasis expansion coefficient calculating unit 32 for calculating aexpansion coefficient O, an object expansion coefficient calculatingunit 33 for calculating an expansion coefficient C, and a spectralreflectance calculating unit 13 for calculating a spectral reflectancep(λ) of an object.

In the spectral reflectance calculating unit 31 having the abovedescribed structure, the object shooting signal Q and reference chartshooting signal G output from the color image input device 1 are inputto the object expansion coefficient calculating unit 33 and the colorchip shooting signal storage unit 11 respectively. In the color chipshooting signal storage unit 11, from a reference chart image made of aplurality of color chips, average values of the shooting signals for allthe color chips are calculated in an appropriate region, and thecalculation result is stored as a color chip shooting signal. In thecolor chip expansion coefficient storage unit 12, expansion coefficientsfor the basis functions of the color chips within a reference color chipobtained in advance are stored.

In the object basis expansion coefficient calculating unit 32, on thebasis of the color chip basis function vn(λ) input from the color chipbasis function storage unit 14 and the object basis functions el(λ)input from the object basis function storage unit 29, the expansioncoefficient O in the case of expanding the object basis function el(λ)by the color basis function vn(λ) is obtained.

In the object expansion coefficient calculating unit 33, on the basis ofthe object shooting signal Q input from the color image input device 1,the color chip shooting signal G input from the color chip shootingsignal storage unit 11, the expansion coefficient B input from the colorexpansion coefficient storage unit 12 and the expansion coefficient Oinput from the object basis expansion coefficient calculating unit 32,the expansion coefficient C (C=(GB⁻¹O)⁻¹Q) in the case of expanding thereflectance of an object by the object basis function el(λ) iscalculated, and the calculation result is output to the spectralreflectance calculating unit 13. The spectral reflectance calculatingunit 13 calculates the spectral reflectance p(λ) of the object from theexpansion coefficient C and the basis functions el(λ) output from theobject basis function storage unit 29.

FIG. 10 shows the structure of the object expansion coefficientcalculating unit 33, which will now be described.

The inverse matrix B⁻¹ of the expansion coefficient B, which iscalculated by the inverse matrix operating unit A34 from the expansioncoefficient B input from the color chip expansion coefficient storageunit 12, the color chip shooting signal G input from the color chipshooting signal storage unit 11, and the expansion coefficient O inputfrom the object basis expansion coefficient calculating unit 32 areinput to the Matrix Multiplication Unit A35, and thus matrix GB⁻¹O iscalculated. The obtained matrix GB⁻¹O is input to the inverse matrixoperating unit B36, and thus an inverse matrix (GB⁻¹O)⁻¹ is obtained,which is further input to the matrix multiplication unit B37.

To the matrix multiplication unit B37, the object shooting signal Q isfurther input. With this input, and the inverse matrix (GB⁻¹O)⁻¹ outputfrom the inverse matrix operating unit B36, the expansion coefficient ofthe object, C=(GB⁻¹O)⁻¹Q is calculated, and the result of thecalculation is output to the spectral reflectance calculating unit 13.

As shown in FIG. 9, in the spectral reflectance calculating unit 13, thespectral reflectance p(λ) of the object is calculated on the basis ofthe expansion coefficient C calculated from the object expansioncoefficient calculating unit 33 and the basis functions el(λ) of theobject, which is output from the object basis function storage unit 29.

FIG. 11 shows the structure of the spectral reflectance calculating unit13, which will now be described.

The spectral reflectance calculating unit 13 includes a basiscoefficient multiplication unit 38 for multiplying the object basisfunctions el(λ) with the corresponding object expansion coefficient C,and a basis function integrating unit 24 for calculating the spectralreflectance.

With the object expansion coefficients C input from the object expansioncoefficient calculating unit 33 and the object basis functions el(λ)input from the object basis function storage unit 29, the basiscoefficient multiplication unit 38 multiples the object basis functionsel(λ) and the object expansion coefficients C=c1 (1=1 to 10)respectively, and the result of the calculation is output to the basisfunction integrating unit 24. The basis function integrating unit 24adds up the data input from the basis coefficient multiplication unit 38with regard to all of the basis functions clel(λ) (1=1 to 10) for eachwavelength, and the result is output to the tristimulus valuecalculating unit 8.

In the tristimulus value calculating unit 8 shown in FIG. 9, thetristimulus value operating unit 16 calculates the tristimulus valuesXYZ under the illumination condition for observing the object on thebasis of the spectral reflectance of the object, calculated in thespectral reflectance calculating unit 13, the observation-siteillumination spectrum data input from the observation-site illuminationspectrum data storage unit 4 and the color matching functions datapre-stored in the color matching function data storage unit 15.

As shown in FIG. 6, the output signal calculating unit 18 converts theXYZ values calculated in the tristimulus value calculating unit 8 intoan RGB signals to be input to the monitor, with use of thecharacteristics data of the CRT monitor, pre-stored in the CRT monitorprofile storage unit 17.

The characteristics data of the CRT monitor includes a matrix M storedin the phosphor tristimulus value data storage unit 25 shown in FIG. 6and gamma curves γ^(R)γ^(G)γ^(B) stored in the RGB gradationcharacteristics data storage unit 26.

The matrix M is a matrix consisting of elements arranged in 3×3, and thegamma curves γ^(R)γ^(G)γ^(B) is a output light intensity to respectiveinput RGB values. The RGB signals output from the RGB value calculatingunit 9 are input to the CRT monitor 6, and the color image of the objectis displayed on the CRT monitor 6.

In the second embodiment, the spectral reflectance of an object isestimated from the object shooting signal Q and reference chart shootingsignal G, shot by the color image input device, the color chip basisfunctions vn(λ) which can expand the spectral reflectance of every colorchip of the reference chart pre-stored, the expansion coefficient B ofeach color chip and the basis function el(λ) of the spectral reflectanceof the object.

As in the case of the first embodiment, the conditions for being able toestimate the spectral reflectance of an object without error are thatthe basis functions el(λ) of the spectral reflectance of an object canbe expanded by the color chip basis functions vn(λ) of the referencechart, and that the color image input device has a certain number ofbands, which is more than the number of basis functions of the spectralreflectances of the object. With these conditions, even if the spectralsensitivities of the color image input device and the illuminationspectrum for shooting are not known, the spectral reflectance of theobject can be estimated.

It should be noted that the second embodiment is different from thefirst one in that the number of bands which is required by the colorimage input device in order to accurately estimate the spectralreflectance of the object, is not the number of color chip basisfunctions, but the number of basis functions of the reflectance of theobject.

In general, the number of basis functions of the spectral reflectance ofthe object is less than the number of basis functions of the spectralreflectances of the color chips, which can expand the basis function ofthe spectral reflectance of the object. Therefore, it becomes possibleto estimate the spectral reflectance of the object by a camera havingthe less number of bands than that of the first embodiment. Further, agreat number of color chips can be used to be able to expand thespectral reflectance of an object at high accuracy, while maintainingthe number of bands of the camera for shooting an object equal to thenumber of basis functions of the spectral reflectance of the object.With use of the same method as in this embodiment, it is possible toaccurately estimate the spectral reflectance of an object such as humanskin, whose spectral reflectance can be reproduced by three basisfunctions at high accuracy, in the case of an ordinary three-banddigital camera, as presented in the fifth embodiment.

FIG. 12 shows the structure of the color image processing apparatusaccording to the third embodiment of the present invention, which willnow be described.

It should be noted that the structural elements of this embodiment whichare equivalent to those shown in FIG. 8 are designated by the samereference numerals, and the detailed explanations therefor will beomitted.

The device of this embodiment includes a color image input device 1 forinputting a color image by shooting or the like, a reference chart 2 aplaced under illumination A, an object 3 a placed under illumination Bdifferent from the illumination A, an observation-site illuminationspectrum data storage unit 4 for storing observation-site illuminationspectrum data, an object basis function storage unit 29 for storing anobject basis function el(λ), an object shooting-site illuminationspectrum data storage unit 39 for storing object shooting-siteillumination spectrum data, which will be explained later, a referencechart shooting-site illumination spectrum data storage unit 40 forstoring reference chart shooting-site illumination spectrum data, animage processing device 41 for estimating a spectral reflectance and anXZY values of an object 3 a from image signals G and Q of the referencechart 2 a and object which are shot, and for outputting RGB signals, anda CRT monitor 6 for displaying the XYZ values of the object under theobservation-site illumination, from the RGB signals.

The input device 1 is equivalent to the input device of the firstembodiment, and a filter turret on which interference filters shown inFIG. 5 are arranged is mounted in the input device 1. Further, thisembodiment has an equivalent structure to that shown in FIG. 7 in termsof the reference chart, and therefore the explanations therefor will notbe repeated. The illumination spectrum data are intensity values ofillumination taken at an interval of 1 nm within a wavelength range of380 nm to 780 nm. The basis functions data of the object are input fromthe object basis function storage unit 29.

The image processing device 41 estimates the spectral reflectance andXYZ values of an object from the above-described inputs, the color chipexpansion coefficient B pre-stored in the image processing device, theexpansion coefficient O′ obtained by expanding the products of the basisfunctions of the object and the object shooting-site illuminationspectrum by the products of the basis functions of the color chips andthe reference chart shooting-site illumination spectrum, the basisfunctions el(λ) of the object and the color matching function data.Further, with reference to the characteristics data of the CRT monitor6, the device 41 outputs the XYZ values of the object under theobservation-site illumination, as RGB signals, to the CRT monitor 6. TheCRT monitor 6 displays the color image of the object on the basis of theRGB signals.

FIG. 13 shows an example of the structure of the image processing device41, which will now be described.

The image processing device 41 consists mainly of a spectral reflectancecalculating unit 42, a tristimulus value calculating unit 8 and anRGB-value calculating unit 9. Here, the structural elements of thisembodiment, which are similar to those shown in FIG. 9, will bedesignated by the same reference numerals, and the detailed descriptionstherefor will not be repeated.

The spectral reflectance calculating unit 42 includes a color chipshooting signal storage unit 11 for storing a color chip shootingsignal, a color chip expansion coefficient storage unit 12 for storingan expansion coefficient B, a color chip basis function storage unit 14for storing color chip basis functions vn(λ) (n=1 to 10), an objectbasis expansion coefficient calculating unit 43 for calculating aexpansion coefficient O′, an object expansion coefficient calculatingunit 44 for calculating an expansion coefficient C′, and a spectralreflectance calculating unit 13 for calculating a spectral reflectancep(λ) of an object from the expansion coefficient C′ and the color chipbasis functions vn(λ) output from the object basis function storage unit29.

The object basis expansion coefficient calculating unit 43 calculates anexpansion coefficient O′ of the case where the products of the objectbasis functions el(λ) and the object shooting-site illuminationspectrum, that is, el(λ)·So(λ), is expanded by the products of the colorchip basis functions vn(λ) and the reference chart shooting-siteillumination spectrum, that is, vn(λ)·Sc(λ).

The object expansion coefficient calculating unit 44 calculates anexpansion coefficient C′ (C′=(GB⁻¹O′)⁻¹Q) on the basis of the objectshooting signal Q, the color chip shooting signal G, the expansioncoefficient B and the expansion coefficient O′.

FIG. 14 shows the structure of the object expansion coefficientcalculating unit 44, which will now be described.

Here, the structural elements of this embodiment, which are similar tothose shown in FIG. 10, will be designated by the same referencenumerals, and the detailed descriptions therefor will not be repeated.

The inverse matrix B⁻¹ of the expansion coefficient B, which iscalculated by the inverse matrix operating unit A34 from the expansioncoefficient B input from the color chip expansion coefficient storageunit 12, the color chip shooting signal G input from the color chipshooting signal storage unit 11, and the expansion coefficient O′ inputfrom the object basis expansion coefficient calculating unit 43 areinput to the matrix multiplication unit A45, and thus matrix GB⁻¹O′ iscalculated. The obtained matrix GB⁻¹O′ is input to the inverse matrixoperating unit B36, and thus an inverse matrix (C=(GB⁻¹O′)⁻¹) isobtained, which is further input to the matrix multiplication unit B37.

To the matrix multiplication unit B37, the object shooting signal Q isfurther input. With this input, and the inverse matrix (GB⁻¹O′)⁻¹ outputfrom the inverse matrix operating unit B36, the expansion coefficient ofthe object, C′ is calculated, and the result of the calculation isoutput to the spectral reflectance calculating unit 13.

As shown in FIG. 13, in the spectral reflectance calculating unit 13,the spectral reflectance p(λ) of the object is calculated on the basisof the expansion coefficient C′ calculated from the object expansioncoefficient calculating unit 44 and the basis functions el(λ) of theobject, which are output from the object basis function storage unit 29.Then, the result of the calculation is output to the tristimulus valuecalculating unit 8.

It should be noted that the spectral reflectance calculating unit 13 hasthe same structure as illustrated in FIG. 11, and therefore theexplanation of this unit will not be repeated.

The tristimulus value calculating unit 8 calculates the tristimulusvalues XYZ under the illumination condition for observing the object 3 aon the basis of the spectral reflectance of the object, calculated inthe spectral reflectance calculating unit 42, the observation-siteillumination spectrum data input from the observation-site illuminationspectrum data storage unit 4 and the color matching functions datapre-stored in the color matching function data storage unit 15. Then,the calculated value is output to the output signal calculating unit 18.

The output signal calculating unit 18 converts the XYZ values calculatedin the tristimulus value calculating unit 8 into RGB signals to be inputto the monitor, with use of the characteristics data of the CRT monitor6, pre-stored in the CRT monitor profile storage unit 17. Thus, thecolor image is displayed on the CRT monitor 6.

In the third embodiment, the spectral reflectance of an object isestimated from the object shooting signal Q and reference chart shootingsignal G, shot by the color image input device under differentillumination conditions, the color chip basis functions vn(λ) which canexpand the spectral reflectance of every color chip of the referencechart pre-stored, the expansion coefficient B of each color chip, thebasis function of the spectral reflectance of the object, the referencechart shooting-site illumination spectrum data and the objectshooting-site illumination data. Here, the conditions for being able toestimate the spectral reflectance of an object without error are thatthe products of the basis functions of the spectral reflectance of theobject and the object shooting-site illumination spectrum, can beexpanded by the products of the color chip basis functions vn(λ) of thespectral reflectance of reference chart and the reference chartshooting-site illumination spectrum, and that the color image inputdevice has a certain number of bands, which is more than the number ofbasis functions of the spectral reflectance of the object.

With these conditions, even if the spectral sensitivities of the colorimage input device are not known, the spectral reflectance of the objectcan be estimated. The third embodiment is different from the second onemainly in the following point. That is, even if the reference chartshooting-site illumination and the object shooting-site illumination aredifferent from each other, the spectral reflectance of an object can beaccurately estimated when the spectra of these illumination are knownand the above conditions are satisfied.

In this embodiment, the color chips image is shot in advance under aknown illumination source and stored. Therefore, it becomes possible toestimate the spectral reflectance of an object when shooting the objectwith use of the same color image input device for shooting the colorchips, without shooting the color chips.

Next, a color image processing apparatus according to the fourthembodiment of the present invention will now be described.

As shown in FIG. 15, the color image processing apparatus includes abasis function storage unit 46 for storing basis functions of thespectral reflectance of human skin, an expansion coefficient settingunit 47 for setting an expansion coefficient corresponding to a basisfunction, a spectral reflectance calculating unit 48 for calculating aspectral reflectance, a spectral reflectance evaluating unit 49 forjudging if a calculated spectral reflectance satisfies an evaluationcondition, and a spectral reflectance storage unit 50 for storing aspectral reflectance which satisfies an evaluation condition.

In the basis function storage unit 46, the basis functions of thespectral reflectance of the human skin, which has values at an intervalof 1 nm within a wavelength range from 380 nm to 780 nm. The number ofbasis functions stored in the basis function storage unit 46 isdetermined in accordance with the contribution rate of the basisfunction obtained by principal component analysis of a great number ofthe spectral reflectances of human skin measured in advance by themeasurement instrument. The M-number of basis functions which arerequired to achieve a contribution rate of 99.9999% or more, are stored.

The expansion coefficient setting unit 47 sets a expansion coefficientCm (m=1 to M) which corresponds to a respective one of the M-number ofbasis functions.

The spectral reflectance calculating unit 48 calculates the spectralreflectance from the data input thereto, that is, the M-number of basisfunctions stored in the basis function storage unit 46 and the M-numberof expansion coefficients set by the expansion coefficient setting unit47.

The spectral reflectance evaluation unit 49, to which the spectralreflectance calculated by the spectral reflectance calculating unit 48is input, judges if the spectral reflectance satisfies the evaluationcondition preset in the spectral reflectance evaluation unit 49. If thecondition is satisfied, the spectral reflectance is output to thespectral reflectance storage unit 50, whereas if the condition is notsatisfied, the expansion coefficient is reset in the expansioncoefficient setting unit 47. This operation is repeated until theM-number of spectral reflectances are output.

Next, the spectral reflectance evaluation unit 49 will now be describedin detail with reference to FIG. 16.

The spectral reflectance evaluation unit 49 includes a non-negativecondition evaluation unit 52 and a preparation condition evaluation unit53. With this structure, the spectral reflectance calculated by thespectral reflectance calculating unit 48 is input thereto, and whetheror not the spectral reflectance has a negative value is examined in thenon-negative condition evaluation unit 52.

Here, in the case where there is no negative value within a wavelengthrange from 380 nm to 780 nm, the spectral reflectance data is sent tothe preparation condition evaluation unit 53, whereas in the case wherethere is a negative value in the range, negative value wavelength datais output to the expansion coefficient setting unit 47.

The preparation condition evaluation unit 53 evaluates whether or notthe spectral reflectance data sent from the non-negative conditionevaluation unit 52 can be designed with available colorant which arestored as a data base in the colorant data storage unit 51 at anaccuracy within an allowable error range.

The colorant data storage unit 51 stores the diffusion coefficient andabsorption coefficient of available colorant in the form of data base.

As the diffusion coefficient and absorption coefficient of a colorant isinput from the colorant data storage unit to the preparation conditionevaluation unit, the spectral reflectance closest to the one which canbe designed from these data is calculated.

In the case where the spectral reflectance sent from the non-negativecondition evaluation unit 52 satisfies an accuracy within an allowableerror range for the spectral reflectance data which can be designed bythese colorant, the spectral reflectance data is stored in the spectralreflectance storage unit 50. In the case where the data cannot bedesigned it is output as design error data to the expansion coefficientsetting unit 47.

The expansion coefficient setting unit 47 resets the expansioncoefficient Cm (m=1 to M) on the basis of the data sent from thenon-negative condition evaluation unit 52 and the preparation conditionevaluation unit 53.

When the color chip having a spectral reflectance prepared in thisembodiment is used as an object whose spectral reflectance is known inthe first to third embodiments, it becomes possible to estimate thespectral reflectance of an object at higher accuracy.

Next, a color image processing apparatus according to the fifthembodiment of the present invention will now be described.

FIG. 17 schematically shows the structure of the embodiment.

In this structure, an object (human image) shot by the digital camera 54and an image of the reference chart 55 prepared for human skin, whichwere taken under the shooting-site illumination are shot as one image.

Then, the shooting image data is written in a memory card 56 providedwithin the digital camera 54. In the memory card 56, basis functions ofspectral reflectances of the color chips of the reference chart 55 whichare read in advance in the digital camera 54 from the chart data recordmemory card attached to the reference chart 55, the expansioncoefficients for the bases of the color chips and the basis functions ofthe spectral reflectances of human skin color are written as header dataof shooting image data. The image data or various data are read from thememory card 56 mounted in the personal computer 57 so as to perform theabove-described process. The data are converted into RGB image data anddisplayed on the monitor 58.

FIG. 18 is a diagram showing the concept of the format of the shootingimage data stored in the memory card 56.

Color chip expansion coefficient data, chart spectral reflectance basisfunction data and object spectral reflectance basis function data arerecorded in the card as header data of one piece of image data. Afterthe header data, the image data is recorded as RGB image data for eachpixel. Each RGB channel is data of 1 byte.

FIG. 19 is a diagram illustrating the concept of the color conversionprocess.

The image data (RGB) signal read from the memory card 56 to the personalcomputer 57 are converted into the spectral reflectance of the objectfrom color chip expansion coefficient data recorded as image headerdata, chip spectral reflectance basis function data, object spectralreflectance basis function data, and chart shooting data.

The spectral reflectance is converted into XYZ values with use ofobservation-site illumination data and color matching functions storedin the computer, and further converted into RGB image data to be inputto a monitor, using the monitor characteristics data, to be displayed asa color image on the CRT monitor.

As described above, the color image processing apparatus of the presentinvention can estimate the spectral reflectance of an object from theobject shooting signal Q and reference chart shooting signal G, shot bythe color image input device under the same illumination condition, thecolor chips basis functions vn(λ) which can expand the spectralreflectances of every color chip of the reference chart pre-stored, theexpansion coefficient B of each color chip, and the basis functions ofthe spectral reflectance of the object. In the present invention, theconditions for being able to estimate the spectral reflectance of anobject without error are that the basis functions of the spectralreflectance of the object can be expanded by the color chip basisfunctions vn(λ) of the reference chart, and that the color image inputdevice has a certain number of bands, which is more than the number ofbasis functions of the spectral reflectances of the object.

With these conditions, even if the spectral sensitivities of the colorimage input device and the shooting-site illumination spectrum are notknown, the spectral reflectance of the object can be estimated.

Thus, according to the present invention, it is possible to provide acolor image processing apparatus for estimating the spectral reflectanceof the object from the color image signal of the object, obtained by theinput device even if the spectral sensitivities of the input device andthe shooting-site illumination spectrum used when the color image signalis input are not known.

Additional advantages and modifications will readily occurs to thoseskilled in the art. Therefore, the invention in its broader aspects isnot limited to the specific details and representative embodiments shownand described herein. Accordingly, various modifications may be madewithout departing from the spirit or scope of the general inventiveconcept as defined by the appended claims and their equivalents.

1. A color image processing apparatus comprising: means for processingan input color image signal of an object in order to reproduce colordata of the object; means for adding, as header data, to the obtainedcolor image signal of the object: (i) a color image signal of areference matter that has a known spectral reflectance, said color imagesignal being obtained using a same lighting condition used inphotographing the object to obtain the color image signal of the object,(ii) spectral reflectance data of the reference matter, and (iii) knowndata of spectral reflectance of the object; and means for storing thecolor image signal with the header data added thereto as image data. 2.The color image processing apparatus according to claim 1, wherein thespectral reflectance data of the reference matter is a basis function ofthe spectral reflectance of the reference matter and an expansioncoefficient for the basis function of the reference matter.
 3. The colorimage processing apparatus according to claim 1, wherein the known dataof spectral reflectance of the object is a basis function of a spectralreflectance of the object.
 4. A color image processing apparatuscomprising: means for processing an input color image signal of anobject in order to reproduce color data of the object; means for adding,as header data, to a color image obtained by photographing the objectand a reference matter having a known spectral reflectance at a sametime in a same frame: (i) spectral reflectance data of the referencematter, and (ii) known data of spectral reflectance of the object; andmeans for storing the color image with the header data added thereto asimage data.
 5. The color image processing apparatus according to claim4, wherein the spectral reflectance data of the reference matter is abasis function of the spectral reflectance of the reference matter andan expansion coefficient for the basis function of the reference matter.6. The color image processing apparatus according to claim 4, whereinthe known data of spectral reflectance of the object is a basis functionof a spectral reflectance of the object.
 7. The color image processingapparatus according to claim 1, wherein the reference matter comprises acolor chart including a plurality of colors.
 8. The color imageprocessing apparatus according to claim 2, wherein the reference mattercomprises a color chart including a plurality of colors.
 9. The colorimage processing apparatus according to claim 3, wherein the referencematter comprises a color chart including a plurality of colors.
 10. Thecolor image processing apparatus according to claim 4, wherein thereference matter comprises a color chart including a plurality ofcolors.
 11. The color image processing apparatus according to claim 5,wherein the reference matter comprises a color chart including aplurality of colors.
 12. The color image processing apparatus accordingto claim 6, wherein the reference matter comprises a color chartincluding a plurality of colors.
 13. An image data storage method foruse in a color image processing apparatus for reproducing color data ofan object, said method comprising: obtaining a color image signal of theobject; obtaining a color image signal of a reference matter having aknown spectral reflectance, by photographing the reference matter usinga same lighting condition used in obtaining the color image signal ofthe object; adding, as header data, to the obtained color image signalof the object: (i) the color image signal of the reference matter, (ii)spectral reflectance data of the reference matter; and (iii) known dataof spectral reflectance of the object; and storing the color imagesignal of the object with the header data added thereto as image data.14. The image data storing method according to claim 13, wherein thespectral reflectance data of the reference matter is a basis function ofthe spectral reflectance of the reference matter and an expansioncoefficient for the basis function of the reference matter.
 15. Theimage data storing method according to claim 13, wherein the known dataof spectral reflectance of the object is a basis function of a spectralreflectance of the object.
 16. The image data storing method accordingto claim 13, wherein the reference matter comprises a color chartincluding a plurality of colors.
 17. An image data storage method foruse in a color image processing apparatus for reproducing color data ofan object, said method comprising: photographing the object and areference matter having a known spectral reflectance at a same time in asame frame to obtain a color image; adding, as header data, to theobtained color image: (i) spectral reflectance data of the referencematter, and (ii) known data of spectral reflectance of the object; andstoring the color image of the object with the header data added theretoas image data.
 18. The image data storing method according to claim 17,wherein the spectral reflectance data of the reference matter is a basisfunction of the spectral reflectance of the reference matter and anexpansion coefficient for the basis function of the reference matter.19. The image data storing method according to claim 17, wherein theknown data of spectral reflectance of the object is a basis function ofa spectral reflectance of the object.
 20. The image data storing methodaccording to claim 17, wherein the reference matter comprises a colorchart including a plurality of colors.